Aiming at the problem that the number of parameters， calculation amount and accuracy of general gesture recognition algorithms are difficult to balance， a lightweight gesture recognition algorithm for basketball referee was proposed. The proposed algorithm was reconstructed on the basis of YOLOV5s （You Only Look Once Version 5s） algorithm： Firstly， the Involution operator was used to replace CSP1_1 （Cross Stage Partial 1_1） convolution operator to expand the context information capturing range and reduce the kernel redundancy. Secondly， the Coordinate Attention （CA） mechanism was added after the C3 module to obtain stronger gesture feature extraction ability. Thirdly， a lightweight content aware upsampling operator was used to improve the original upsampling module， and the sampling points were concentrated in the object area and the background part was ignored. Finally， the Ghost-Net with SiLU （Sigmoid Weighted Liner Unit） as the activation function was used for lightweight pruning. Experimental results on the self-made basketball referee gesture dataset show that the calculation amount， number of parameters and model size of this lightweight gesture recognition algorithm for basketball referee are 3.3 GFLOPs， 4.0×10⁶ and 8.5 MB respectively， which are only 79%， 44% and 40% of those of YOLOV5s algorithm， mAP@0.5 of the proposed algorithm is 91.7%， and the detection frame rate of the proposed algorithm on the game video with a resolution of 1 920×1 280 reaches 89.3 frame/s， verifying that the proposed algorithm can meet the requirements of low error， high detection rate and lightweight.